Low-Cost, High-Performance Barrier Synchronization on Networks of Workstations

structions between synchronizations). Barriers are important synchronization operations in multiprocessor systems [5, 19]. When a processor executes a barrier instruction, it first checks in at the barrier to indicate to the other processors that it has arrived at the specified synchronization point. It then must wait for all other processors participating in the barrier to check in, after which all processors can proceed past the barrier to begin executing their next assigned tasks. Tasks may include parallel loop iterations [16], emulation of very long instruction words (VLIW) [14], or dataflow steps [11]. Current barrier implementations typically use software trees [18], softwareor hardware-based counters [12, 13, 17], or hardware fan-in trees [3, 5, 6, 19, 20]. While the software trees are inexpensive, their synchronization delay is too long for fine-grained applications, especially on systems lacking a shared-bus. Counter-based methods introduce potentially high contention to access the shared counters, which can lead to unacceptably long, and unpredictable, synchronization delays. However, the best hardware solutions are not easily adapted to a wide range of existing systems, but depend on a particular topology, or even a particular implementation. The fan-in tree is an example of a class of hardware solutions with very good performance. The hardware required to implement the best techniques is often complex and expensive, requiring O(N 2) wiring complexity [5, 20] for the fastest performance, a severe problem on networks of workstations. Slow and expensive associative memory may also be required [20]. Note that fine granularity precludes using barriers in a multiprogramming environment since a taskswitch by one PE would delay any barrier(s) in which it is involved until that PE resumes execution. Extremely fast context-switching [1, 8] can be used to mitigate this problem. The barrier mechanism proposed in this paper is inexpensive, using only simple bit-serial hardware in each PE with only a single-conductor serial ring between PEs. Since the ring itself has no clocked sequential logic, the only latency introduced by the barrier hardware is a single gate JOURNAL OF PARALLEL AND DISTRIBUTED COMPUTING 40, 131–137 (1997) ARTICLE NO. PC961273